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From  the  Director 


should  we  care  where  good  M&S  data  goes?  And,  by  extension,  where  does  good  information  and  good 
knowledge  go?  Because  if  it  is  really  good,  we  want  to  be  able  to  use  the  data,  information,  and  knowledge  over  and  over 
again.  As  a  parallel  to  that  old  saw  about  how  to  get  to  Carnegie  Hall,  we  believe  that  how  you  get  to  simulation  success 
is  reuse,  reuse,  reuse.  Why?  Because  reuse  is  the  key  to  improving  results,  lowering  risk,  enhancing  efficiency,  and  saving 
money  on  modeling  and  simulation.  And  how  do  we  get  to  reuse?  By  storing  the  data  is  a  way  that  promotes  discovery, 
access,  evaluation,  and  adaption/adoption. 


The  discovery,  access,  and  evaluation  of  data,  models,  or  tools  are  all  decidedly  non-trivial  for  current  members  of  the 
DoD  M&S  community.  But  carefully  designed  repositories,  portals,  wikis,  and  the  Global  Information  Grid  (GIG)  can  help. 


This  issue  of  the  MSI  AC  Journal  presents  three  extremely  informative  articles  about  M&S  data.  The  paper  by  Feinberg, 
Misch,  and  Talbot,  "M&S  Repositories  -  Lessons  Observed",  expounds  on  the  theme  of  what  makes  the  storage  of  M&S 
information  effective  and  feasible.  It  also  highlights  new  and  emerging 
approaches  to  enhancing  the  discovery  and  reuse  of  data.  The  article  by 
Colon,  Tran,  Yao,  Anhalt,  and  Curiel, "Modeling  Situational  Awareness  Using 
Scenario  Objects  and  Events  for  Constructive  Experimentation"  describes  how 
good  data  and  information  can  be  applied  on  the  battlefield  through 
improved  situational  awareness.  The  article  by  Leon-Barth  and  DeMara, 

"Network  Communications  Effects  Simulator  Evaluation  Scenarios  for 
Joint  Tactical  Radio  System  (JTRS)  and  Warfighter  Information  Network-Tactical 
(WIN-T)"  is  an  excellent  example  of  how  data  is  used  in  M&S  to  represent  the 
effectiveness  of  new  and  emerging  systems. 

Together,  these  papers  examine  and  enlighten  us  on  how  good  data  should  be 
kept,  maintained,  and  enhanced.  And  this  issue  of  the  MSIAC  Journal  itself 
contributes  directly  to  our  theme:  keeping  good  data  in  the  spotlight.  We  are 
all  fighting  to  avoid  the  worse  possible  destination  for  our  good  data:  the 
metaphorical"Timbuktu"as  represented  by  the  gigantic  government 
warehouse  depicted  near  the  end  of  "Raiders  of  the  Lost  Ark". 

Dane  Mullenix,  MSIAC  Director 
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M&S  Repositories  -  Lessons  Observe< 

Written  by:  Dr.  Jerry  M.  Feinberg, 

Gary  L.  Misch, 

Laurie  H.  Talbot 


ABSTRACT :  Although  there  is  a  plethora  of  repositories 
in  general ',  and  many  M&S  repositories  in  specific ;  their 
advantages  to  members  of  the  M&S  community  are  not  clear. 
There  is  little  hard  evidence  on  repositories'  usefulness.  There 
is  less  documentation  on  their  management.  And  there  is 
almost  no  information  on  their  costs. 

The  Modeling  and  Simulation  Information  Analysis  Center 
(MSIAC)  has  been  involved  in  operating ;  managing , 
maintaining ;  and  reviewing  repositories  since  its  inception  in 
1 999.  This  paper  summarizes  some  lessons  observed  over  this 
time  period  on  the  usefulness  of  M&S  repositories;  what  makes 
them  successful  (or  not);  and  their  ability  to  offer  a  potential 
cost ;  schedule ;  and  risk  advantage  to  an  M&S  developer  or 
user. 

The  MSIAC  has  noted  that  a  small,  controlled  repository  is  more 
likely  to  succeed,  that  a  user-populated  repository  requires 
extreme  discipline  in  the  community  or  else  political  clout  to 
succeed,  and  that  a  large  and  broadly-based  M&S  repository 
will  underperform  due  to  policy  and  business  model  issues. 
These  observations,  and  others,  are  presented  in  an  evaluation 
framework  for  a  repository  that  includes  the  following  fields: 
What  is  its  justification?  What  policy  supports  its  use?  What 
is  its  scope?  Who  will  be  allowed  access  to  it?  How  is  it 
implemented,  namely  who  populates  it,  who  validates  the 
information,  who  manages  it,  and  who  pays  for  it? 

The  paper  concludes  with  a  section  discussing  the  future 
of  M&S  repositories.  One  special  topic  is  the  desirability  of 
implementing  an  M&S  repository  with  Wiki  software. 

Keywords: 

Repositories,  Cost  Effectiveness,  Reuse 


1.  Introduction 

M&S  practitioners  have  used,  and  continue  to  use,  M&S 
repositories  to  store  and  find  simulation  components  and 
data.  However,  the  specific  utility  of  these  repositories 
to  members  of  the  M&S  community  is  not  clear.  There 
is  little  hard  evidence  on  repositories7  usefulness,  their 
management,  or  their  costs. 

The  Modeling  and  Simulation  Information  Analysis  Center 
(MSIAC),  the  Department  of  Defense  (DoD)  IAC  dedicated 
to  supporting  M&S,  has  been  involved  in  operating, 
managing,  maintaining,  and  reviewing  repositories  since 
its  inception  in  1999.  This  paper  analyzes  information 


collected  by  the  MSIAC  on  the  usefulness  of  M&S 
repositories.  The  paper  also  summarizes  lessons  observed 
over  this  time  period  about  what  makes  a  repository 
successful.  This  paper  also  presents  these  observations 
in  an  evaluation  framework  for  a  repository  that  includes 
fields  for  repository  justification,  policy  support,  scope, 
access,  and  implementation  (namely  population, 
validation,  management,  and  funding). 

The  paper  also  discusses  the  future  of  M&S  repositories 
and,  in  particular,  the  desirability  of  implementing  an  M&S 
repository  with  Wiki  software. 

We  note  that  this  paper  presents  only  the  lessons 
observed  by  the  authors.  We  suspect  that  there  are  also 
many  additional  lessons  either  not  yet  learned  or  not  yet 
observed. 


2.  Background  on  Repositories 

Quoting  the  current  default  repository  of  knowledge, 
Wikipedia  [1],  a  repository  is: 

"a  place  where  data  are  stored  and 
maintained.  A  repository  can  be  ...  a 
place  where  data  are  stored, ...  a  place 
where  anything  is  stored  for  probable 
reuse, ...  a  place  to  store  digital  data." 

This  is  distinguished  from  a  depository,  a  place  just  for 
storing  things  (e.g.,  the  United  States  Bullion  Depository, 
also  known  as  Fort  Knox),  by  the  desire  for"probable  reuse." 

Repositories  are  desirable  because  they  allow  permitted 
members  of  a  community  to  discover,  obtain,  and  use/reuse 
resources.  Properly  implemented  repositories  can  then 
enhance  efficiency,  standardization,  and  cost  savings. 

The  downside  of  repositories  often  is  the  lack  of  a  business 
model  encouraging  their  population,  use,  and  content 
sharing.  For  example,  developers  can  be  reluctant  to  share 
their  resources;  data  consumers  are  often  averse  to  using 
other's  data;  and  owners  typically  want  to  know  who  will 
use  their  resources,  and  for  what  purpose,  before  sharing  [2]. 

For  example,  if  a  program  uses  someone  else's  resource, 
the  manager  is  taking  the  chance  that  the  resource  is 
unsuitable  for  the  program's  purposes,  and  hence  might 
"be  burned"  by  using  it.  Similarly,  from  many  program 
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managers' views,  it  probably  costs  less  to  build  a  new 
resource  than  to  ensure  that  someone  else's  resource  is 
suitable  for  a  particular  use. 

One  more  interesting  issue  with  repositories  is  why 
have  them  at  all  when  one  can  just  use  a  modern 
internet  search  engine  (e.g.,  Google  [3])  to  access  almost 
everything  that  is  needed.  When  a  user  employs  a  search 
engine,  the  user  takes  primary  responsibility  for  assessing 
the  value  and  suitability  of  the  information  attained.  If 
everyone  in  a  community  uses  this  approach,  then  the 
assessments  are  repeated  multiple  times  resulting  in  gross 
inefficiency  and  extra  cost.  The  value  of  a  repository  is 
that  the  assessment  is  performed  once  and  then  others 
rely  on  it  (to  a  greater  or  lesser  degree,  depending  on 
the  repository).  Thus  the  repository's  greater  difficulty  of 
individual  resource  discovery  can  be  outweighed  by  the 
group  savings  on  time  and  effort. 


3.  M&S  Repository  Examples 

This  section  presents  three  important  examples  of  M&S 
repositories. 

3.1  MSRR 

One  well-known  M&S  repository  is  the  Modeling  and 
Simulation  Resource  Repository,  or  MSRR.  The  MSRR  was 
first  conceived  in  1 993  as  the  Modeling  and  Simulation 
Data  Repository.  As  the  anticipated  scope  of  simulation 
interoperability  increased,  it  became  clear  that  defense 
simulations  would  require,  and  could  reuse,  many  more 
resources  than  just  databases.  The  models  and  simulations 
themselves  became  candidates  for  inclusion,  as  well  as 
data  transformation  tools,  VV&A  histories,  use  histories,  etc. 

In  a  period  of  diminishing  budgets,  as  perceived 
duplication  of  effort  ("continually  re-inventing  the  wheel") 
was  escalating  costs  to  a  politically  unacceptable  level, 
a  robust  reuse  strategy  was  developed  to  demonstrate 
commitment  to  cost  effective  policies.  To  stave  off 
program  cancellation  due  to  lack  of  funds,  program 
managers  were  expected  to  share  their  resources  and 
populate  the  MSRR  as  a  matter  of  survival. 

The  MSRR  initially  came  on  line  in  late  1 995.  Due  to 
management  and  policy  issues,  the  Services  brought 
their  own  MSRRs  on  line,  along  with  the  National  Ground 
Intelligence  Center  and  the  Missile  Defense  Agency. 


The  original  concept  for  the  MSRR  was  that  of  a  true 
repository  containing  actual  simulation  components 
and  datasets.  However,  a  variety  of  policy  factors  led  to 
providing  only  a  catalogue  of  resources.  Thus  the  current 
MSRR  System  provides  retrieval  of  metadata  descriptions 
only  of  modeling  and  simulation  resources.  Providers 
include  the  DoD  system,  Army,  Navy,  Air  Force,  and  the 
Defense  Intelligence  Agency. 

Originally,  the  MSRR  user  base  was  to  encompass  all 
simulation  users.  These  ranged  from  the  highest  level 
planners  to  the  lowest  level  tactical  operators,  and 
from  major  project  systems  engineers  to  individual 
programmers  building  simple  models.  This  large 
scope  was  a  significant  driver  in  both  the  repository's 
functionality  and  anticipated  content  [4]. 

3.2  OAM L 

The  Navy's  Chief  of  Naval  Operations  (CNO)  established 
the  Oceanographic  and  Atmospheric  Master  Library 
(OAML)  in  1984.  OAML  contains  ocean  models, 
electromagnetic  (EM)  models,  acoustic  models, 
meteorological  models,  and  a  category  called  other 
models.  OAML  also  contains  ocean  databases, 
meteorological  databases,  acoustic  databases,  and 
electromagnetic  databases.  The  OAML  resources  support 
the  Department  of  the  Navy,  Department  of  Defense, 
research  and  development  laboratories  and  Joint  and 
NATO  activities  with  state-of-the-art  Navy-standard 
products. 

OAML  was  developed  to  provide  consistency  and 
standardization  for  all  oceanographic  and  meteorological 
programs  used  by  the  Navy.  This  was  necessary  since  the 
Navy  has  developed  and  used  multiple  oceanographic 
and  atmospheric  models  and  databases.  One  way  to 
guarantee  consistency  in  simulation  outputs  was  through 
policy  requiring  use  of  the  resources  in  OAML,  which  is 
now  the  Navy  standard  library  for  meteorological  and 
oceanographic  databases,  models,  and  algorithms. 

The  responsibility  for  maintaining  the  models  and 
databases  in  OAML  rests  with  the  Naval  Oceanographic 
Office  (NAVO)  located  at  the  Stennis  Space  Center, 
Mississippi.  Commander,  Naval  Meteorology  and 
Oceanography  Command  is  designated  as  the  OAML 
Configuration  Manager. 

General  descriptions  of  the  various  oceanographic  and 
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atmospheric  models  and  databases  are  provided  in  the 
"Oceanographic  and  Atmospheric  Master  Library  (OAML) 
Summary"  published  by  NAVO.  This  documentation  also 
delineates  the  applications  and  limitations  of  the  OAML 
models  and  databases. 

OAML  started  in  1 984  with  seven  databases.  In  2004,  it 
contained  35  models,  1 9  databases,  and  8  algorithms,  or  a 
total  of  62  resources  [5],  [6],  [7],  [8]. 

3.3  ACAT  I  IDEs 

A  valuable  class  of  repositories  is  the  Integrated  Digital 
Environments  (or  IDEs)  that  have  been  developed  for  most 
recent  ACAT  I  procurements  (also  known  as  Major  Defense 
Acquisition  Programs).  These  are  small,  controlled 
repositories  with  carefully  selected  M&S  and  data 
components  with  a  specific  purpose  in  support  of  a  single 
program.  Integrated  Digital  Environments  are  also  called 
Integrated  Data  Environments  or  Integrated  Development 
Environments. 

From  the  Defense  Acquisition  Guidebook  [9],  DoD  policy 

"requires  the  maximum  use  of  digital 
operations  throughout  the  system  life 
cycle.  ...  Program  managers  should 
establish  a  data  management  system 
within  the  IDE  that  allows  every 
activity  involved  with  the  program  to 
cost-effectively  create,  store,  access, 
manipulate,  and  exchange  digital  data. 

This  includes,  at  minimum,  the  data 
management  needs  of ...  modeling  and 
simulation  activities, ..." 

One  excellent  example  of  an  IDE  is  that  developed  for  the 
Joint  Strike  Fighter  (JSF)  program  [10],  [11]. 


4.  Repository  Evaluation  Framework 

The  possibility  of  cost  savings,  re-use,  and  efficiency 
all  indicate  that  repositories  can  be  of  value  to  the 
communities  they  serve.  While  it  is  quite  easy  to  state  that 
one  repository  has  been  a  success,  and  another  has  been  a 
failure,  it  is  very  hard  to  produce  a  basis  for  such  statements. 

We  believe  that  declarations  of  success  or  failure  without 
definitive  measures  are  virtually  useless.  Consequently, 
this  section  of  the  paper  offers  a  framework  that  supports 
a  repository  evaluation.  The  framework  encompasses  five 
distinct  evaluation  areas,  or  measures,  each  of  which  is 


described  below. 

The  top-most  measure  we  call  "justification."  This  is 
concerned  with  the  fundamental  community  need  that 
rationalizes  the  establishment  of  the  repository.  This 
qualitative  measure's  values  could  include  cost  savings, 
re-use,  efficiency,  or  standardization. 

A  second  measure  is  "policy  support."  This  is  concerned 
with  policies  that  mandate  the  population,  use,  and 
upkeep  of  the  repository.  Values  include  existence  and 
enforceability. 

A  third  measure  is  "scope."  This  involves  the  contents  of 
the  repository  and  specifically  how  much  is  included  in 
it.  Values  include  the  number  of  resource  types  and  the 
overall  number  of  resources. 

The  fourth  measure  is  "access."  This  is  concerned  with  the 
authorized  users  of  the  repository  -  many  or  few.  If  the 
cost  of  replicating  certain  resources  is  high,  a  repository 
need  not  have  a  large  number  of  users  to  provide  value.  A 
secondary  issue  is  the  difficulty  of  accessing  the  contents. 

The  fifth  measure  is  "implementation."  This  measure  has 
several  dimensions.  These  include  how  the  repository 
is  populated,  how  the  information  contained  in  the 
repository  is  validated,  how  the  repository  is  managed, 
and  how  the  repository  is  funded. 


This  information  is  summarized  below  in  Table  4.1 . 


Repository  Evaluation  Framework 

Measure 

Metric  Values 

Justification 

Cost  savings,  time  savings,  risk  reduction 
through  re-use,  standardization 

Policy  Support 

Existing,  enforceable 

Scope 

Wide,  limited 

Access 

Open,  restricted 

Implementation: 

Population, 

Validation, 

Management, 

Funding 

Single  person,  resource  owners, 
programs, ... 

Table  4.1  -  Basic  Framework 


5.  Evaluation  of  Selected  M&S  Repositories 
Using  the  Framework 

We  can  test  the  framework  proposed  above  by  using  it  to 
evaluate  the  three  example  M&S  repositories. 

We  stress  here  that  these  evaluations  are  the  opinions  of 
the  authors  only. 
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5.1  MSRR 


5.3  ACAT  I  IDE 


Table  5.1  contains  our  evaluation  of  the  MSRR  Table  5.3  contains  our  evaluation  of  an  ACAT  I  IDE  in 

in  terms  of  the  repository  evaluation  framework.  terms  of  the  repository  evaluation  framework. 


Repository  Evaluation  for  MSRR 

Measure 

Metric  Values 

Justification 

Cost  savings,  time  savings,  risk  reduction 
through  re-use 

Policy  Support 

Inconsistent  -  currently  weak  to  non- 
existents 

Scope 

Large  (virtually  unrestricted) 

Access 

Theoretically  wide  but  practically 
hindered  by  implementation  factors 

Implementation: 

Population, 

Validation, 

Management, 

Funding 

Population  -  originally  performed  by 
resource  owners  (was  ineffective)  and 
currently  performed  by  MSIAC. 

Validation  -  performed  by  resource  users. 

Management  -  MSIAC  but  without  any 
clear  direction.  Funding  -  no  recent 
funding  for  maintenance  or  update. 

Table  5.1  -  Evaluation  for  the  MSRR 


5.2  OAML 


Table  5.2  contains  our  evaluation  of  the  OAML 
in  terms  of  the  repository  evaluation  framework. 


Repository  Evaluation  for  OAML 

Measure 

Metric  Values 

Justification 

Standardization,  cost  savings,  time 
savings,  risk  reduction  through  re-use 

Policy  Support 

Clear  and  enforceable 

Scope 

Service-wide,  limited  to  specific  topics 

Access 

Limited  to  designated  users 

Implementation: 

Population, 

Validation, 

Management, 

Funding 

Population,  validation,  management, 
funding  -  Navy  and  the  Program  Office 

Table  5.2  -  Evaluation  for  OAML 


Repository  Evaluation  for  an  ACAT  1  IDE 

Measure 

Metric  Values 

Justification 

Cost  savings,  time  savings,  risk  reduction 
through  re-use,  standardization 

Policy  Support 

Clear  and  enforceable 

Scope 

Program-wide,  limited  to  specific  topics 

Access 

Limited  to  designated  users 

Implementation: 

Population, 

Validation, 

Management, 

Funding 

Population,  validation,  management, 
funding  -  the  ACAT  1  program 

Table  5.3  -  Evaluation  for  an  ACAT  I  IDE 


6.  Lessons  Observed  and  Issues  for 
M&S  Repositories 

This  section  summarizes  some  lessons  observed  by  the 
authors  over  the  course  of  their  involvements  with  M&S 
repositories.  We  find  it  easiest  to  categorize  these  lessons 
using  the  framework  for  repository  evaluation. 

6.1  Lessons  observed  -  Justification 

Apparently,  it  has  been  quite  easy  to  justify  the 
development  of  M&S  repositories.  One  standard  method 
is  the  "if  you  build  it,  he  will  come"  [12]  approach.  This 
has  not  proven  true  for  the  MSRR,  as  the  system  seems 
to  be  very  under-utilized.  The  wide-scope  MSRR  can  be 
justified  on  several  grounds,  primarily  reuse,  but  the  issues 
of  return  on  investment  or  cost  savings  are  shakier  due  to 
the  low  utilization. 

In  other  cases,  this  justification  has  proven  out.  A  shining 
example  is  OAML  where  the  desire  for  standardization 
across  the  Navy  M&S  community  largely  has  been 
attained.  Standardization  and  control  leading  to  reuse 
also  has  justified  the  ACAT  I  IDEs. 


6.2  Lessons  Observed  -  Policy  Support 


It  is  quite  clear  that  a  strong  policy  mandating  both  the 
use  and  the  population  of  a  repository  is  a  main  key  to 
success. 
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Such  a  policy  is  easier  to  develop  and  implement  if  the 
scope  of  the  repository  is  not  too  large  since  the  number 
of  affected  users  is  more  limited  and  they  are  then  easier 
to  control. 

The  MSRR  could  have  been  much  more  effective  if  there 
had  been  strong  accompanying  policy  requiring 

developers  to  supply  the  metadata  for  their  resources, 
however  the  span  of  management  authority  within 
the  DoD  makes  implementation  of  such  strong  policy 
problematic.  This  holds  true  for  all  of  the  nodes  of  the 
MSRR. 

6.3  Lessons  Observed  -  Scope 

Our  review  indicates  that  the  wider  the  scope  of  the 
repository,  the  more  difficult  it  is  to  make  it  a  success. 

Part  of  the  reason,  discussed  above,  is  because  it  is  harder 
to  implement  supporting  policy.  Another  part  is  that 
it  is  harder  to  maintain  control  over  a  larger  number  of 
resources  and  another  part  is  the  cost  -  the  greater  the 
number  of  resources,  the  more  the  cost  to  populate 
and  update  the  repository.  Finally,  attempting  to  serve 
many  different  types  of  users  increases  the  repository 
requirements,  consequently  complicating  the  design  and 
subsequent  implementation,  leading  to  higher  costs. 

6.4  Lessons  Observed  -  Access 

We  have  not  seen  a  strong  correlation  between  the 
success  of  a  repository  and  the  scope  of  allowable  users. 
What  seems  to  be  much  more  important  is  the  ease  of 
access  to  the  repository.  OAML  users  require  special 
approval,  but  seem  to  obtain  this  easily.  ACAT I  IDE 
users  are  quickly  authorized  by  their  programs.  In  both 
cases,  repository  effectiveness  is  probably  improved  by 
having  remote  access  available  with  associated  security 
procedures  in  place. 

The  reported  ease  of  use  of  the  MSRR  varies.  Many 
potential  users  are  confused  by  the  fairly  archaic  interface. 
Yet  many  experienced  engineers  have  no  problems  with 
the  system. 

Finally,  the  policy/design  underlying  the  MSRR  has 
permitted  metadata  only  to  be  stored.  Consequently, 
discovering  the  resource  in  the  MSRR  is  not  equivalent 
to  attaining  the  resource.  Without  an  effective  business 
model  encouraging  sharing,  a  metadata  repository  will 
not  succeed. 


6.5  Lessons  Observed  -  Implementation 

For  populating  resources  into  the  repository,  the  method 
utilized  by  the  MSRR  is  totally  ineffective.  This  method 
relies  on  the  resource  owner's  "good  will"  to  take  the 
steps  required  to  insert  the  resource.  This  has  failed  on 
several  counts.  The  first  is  the  time  and  effort  (hence  cost) 
involved.  The  second  is  that  publication  of  a  resource's 
existence  with  a  point  of  contact  leads  to  frequent 
communications  that  can  require  significant  time  and 
effort  on  the  contact's  part.  The  third  is  a  potential  loss 
of  control  of  the  model's  code  since  the  MSRR  relies  on 
the  resource  owner  for  configuration  management  but 
changes  to  the  code  could  be  made  by  any  third  party 
user.  A  fourth  is  a  perception  by  several  owners  that  only 
they  know  how  to  execute  their  particular  simulations 
properly  as  it  is  thought  to  be  more  of  an  "art"  than  a 
"science" to  run  them. 

OAML  and  the  ACAT  I  IDEs  effectively  pay  for  the 
population  of  their  repositories.  There  is  also  strong  policy 
(by  the  Navy  or  by  the  program  office)  requiring  use  of 
their  repositories.  This  has  been  effective. 

Validation  and  configuration  management  of  MSRR 
resources  is  performed  by  the  resource  owners.  Validation 
of  OAML  models  is  performed  by  the  OAML  program. 
Validation  of  ACAT  I  IDE  models  is  performed  (at  least  in 
theory)  by  the  program  office.  We  have  not  observed  any 
problems  with  these  methods. 

Management  of  OAML  and  of  an  ACAT  I  IDE  is 
straightforward  and  executed  by  the  program  offices. 
Management  of  the  MSRR  has  been  inconsistent, 
fractured,  and  at  times  chaotic.  This  inconsistent 
management  of  the  MSRR  has  led  to  reduced  usage  and 
has  also  precluded  the  development  of  effective  policies. 

Funding  of  repositories  is  always  an  issue.  The 
management  of  a  repository  has  to  justify  the  program  to 
maintain  its  budget.  Expanding  the  scope  of  a  repository 
demands  a  higher  budget.  OAML  and  an  ACAT  I  IDE 
live  within  their  fluctuating  budgets  by  constraining  the 
number  of  resources  and  some  of  the  associated  services 
provided.  In  fact,  there  are  many  examples  of  building 
very  effective  repositories  on  a  tight  budget  by  choosing 
to  constrain  the  number  of  resources  being  managed. 

6.6  Summary  of  Lessons  Observed 

So,  in  summary,  it  first  appears  that  a  small,  controlled 
repository  is  more  likely  to  succeed. 
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Second,  a  user-populated  repository  requires  extreme 
discipline  in  the  community  or  else  political  clout  to 
succeed.  Third,  a  large  and  broadly-based  repository  may 
underperform  due  to  policy  and  business  model  issues. 
Fourth,  while  incorporating  a  broad  user  base  may  appear 
to  be  wise,  constraining  the  user  base  from  which  initial 
requirements  are  derived  can  lead  to  a  more  affordable 
system  without  necessarily  impacting  scalability. 


7.  Future  for  M&S  Repositories 

How  should  repositories  for  models  and  simulations 
change  to  be  more  effective,  efficient,  and  affordable?  In 
our  view,  the  answer  to  this  question  centers  on  the  major 
evaluation  criteria  in  our  framework. 

7.1  Suggestions 

Justification:  there  is  a  continuing  need  for  access  to 
modeling  and  simulation  components  for  adaption  or 
adoption,  and  to  support  standardization.  Repositories 
are  here  to  stay  provided  that  their  contents  are  properly 
validated  and  reusable  so  as  to  compete  cost  effectively 
with  internet  search  engines.  The  initial  step  always  is  to 
determine  who  the  users  of  a  repository  are  going  to  be 
and  what  these  users  are  going  to  want  (requirements). 

Policy  support:  future  repositories  will  succeed  when 
supported  by  strong  policies  in  two  areas.  The  first  is  to 
mandate  the  use  of  modeling  and  simulation  components 
that  are  contained  in  the  repository,  whether  for  adoption, 
or  adaption,  or  supporting  new  development.  The  second 
is  to  require  the  population  of  these  repositories  with  a 
set  of  M&S  components  satisfying  important  selection 
criteria.  These  policies  need  to  be  enforceable. 

Scope:  future  repositories  have  a  greater  likelihood  of 
success  if  they  are  narrower  in  scope  and  controlled.  Thus 
we  suggest  a  set  of  separate,  constrained,  limited-scope 
repositories  that  are  tightly  controlled,  but  linked  so  that 
discovery  across  the  entire  set  is  possible. 

Access:  future  repository  or  repositories  should  allow 
wide  access  provided  this  does  not  drive  the  architecture/ 
design  to  a  difficult-to-maintain,  costly-to-implement 
solution.  The  design  requirements  should  be  derived 
from  a  smaller,  cohesive  set  of  users.  Certain  resources 
can  always  be  protected  passwords  or  other  means  if 
necessary. 

Implementation:  future  repository  population  should 
be  controlled  by  the  repository  manager/owner  who  is 
also  responsible  for  validating  the  information  contained 
therein.  These  individual  manager/owners  should  also 
pay  for  development,  population,  and  upkeep.  It  does  not 


matter  if  the  repository  is  distributed  across  various  nodes 
provided  that  it  is  presented  to  the  user  as  one  logical 
repository. 

7.2  Other  Approaches 

As  discussed  earlier,  relying  on  internet  search  engines 
presents  a  viable  possibility  for  replacing  repositories,  but 
these  engines  introduce  difficulties  of  their  own.  A  more 
interesting  approach  is  developing  a  Wiki-based  repository 
[1 3]  in  which  users  or  third  parties  populate  the  repository 
with  information  about  resources.  This  approach  eliminates 
the  bottleneck  of  relying  upon  a  specified  entity  or  group 
to  perform  all  the  work  of  population  and  "spreads  the  costs 
around."  However,  it  can  lead  to  free-for-all  battles  on  the 
information  content  with  competing  views  of  a  resource 
playing  out  in  public. 

Internally,  the  MSIAC  has  been  testing  a  Wiki-based 
repository  for  some  of  its  own  resources.  In  this  case,  the 
"editors"  of  the  resources  are  a  limited  group  and  the  scope 
of  the  repository  is  controlled.  We  hope  to  report  more 
details  of  this  approach  in  a  later  paper. 

Finally,  amongst  all  of  the  hoopla  and  teeth  gnashing 
concerning  repositories,  we  suggest  heeding  the  words  of  a 
famous  British  knight: 

". .  .you  can't  always  get  what  your  want, 
but  if  you  try  sometimes,  you  just  might 
find,  you  get  what  you  need"  [1 4] 


8.  Acronyms 


ACATI 

Acquisition  Category  1  -  Major  Defense  Acquisition  Programs 

CNO 

The  Chief  of  Naval  Operations 

DoD 

Department  of  Defense 

EM 

Electromagnetic 

GIG 

Global  Information  Grid 

IAC 

Information  Analysis  Center 

IDE 

Integrated  Data  Environment 

JSF 

Joint  Strike  Fighter 

M&S 

Modeling  and  Simulation 

MSIAC 

Modeling  and  Simulation  Information  Analysis  Center 

MSRR 

Modeling  and  Simulation  Resource  Repository 

NAVO 

Naval  Oceanographic  Office 

OAM  L 

Oceanographic  and  Atmospheric  Master  Library 

SPM 

Smart  Product  Model 

VV&A 

Verification,  Validation,  and  Accreditation 
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ABSTRACT:  Various  simulators  have  been  identified 
as  Communications  Effects  Servers  (CES)  to  predict  the 
bandwidth  necessary  to  support  Future  Combat  Systems 
(FCS)  training  exercises.  This  paper  examines  features  ofCES's 
within  particular  Future  Combat  Systems  communication 
simulation  scenarios  that  are  defined  to  serve  as 
benchmarks.  The  scenarios  are  designed  to  evaluate  the 
demands  of  FCS  as  part  of  the  Objective  Force  (OF),  providing 
results  that  can  be  measured  and  used  to  correlate  results 
between  different  CES  technologies  and  verify  their  accuracy. 

At  present,  the  Caspian  Sea  is  the  accepted  test  terrain 
database  for  FCS/OF  communications  [  1  ].  The  outcome 
generated  by  such  scenarios  can  produce  reliable 
information  that  can  be  used  to  compare  the  fidelity  of  the 
FCS  communications  for  a  defined  environment.  Moreover, 
the  results  provided  from  each  of  these  models  can  be 
used  to  evaluate  communications  capacity  requirements 
of  small  FCS  exercises  including  other  elements  such  as 
embedded  training  that  is  suspected  to  increase  the  existing 
FCS  communications  demands  during  certain  stages  of  the 
simulation. 

Overview  of  FCS  Network 

Currently  Future  Combat  Systems  (FCS)  contractors 
propose  an  alternative  FCS  protocol  that  will  integrate 
JTRS  and  WIN-T  into  FCS.  The  protocol  System-of- 
Systems  Common  Operating  Environment  (SOSCOE) 
which  is  under  development  has  the  goal  of  enabling 
straightforward  integration  of  separate  software 
packages,  independent  of  their  location,  connectivity 
mechanism  and  the  technology  used  to  develop  them 
[2].  FCS  will  employ  an  integrated  computer  system  to 
host  SOSCOE,  support  networking,  and  employ  consistent 
data  storage/retrieval  across  all  FCS  cells  and  applications. 
Nevertheless,  SOSCOE  is  not  included  as  part  of  any  of  the 
CES  simulators  currently  available. 

FCS  is  intended  to  provide  the  core  block  of  the  US 
Army's  Future  Force.  The  FCS  Family-of-Systems  (FoS) 
is  to  be  coupled  to  the  C4ISR  network  by  a  multilayered 
Communications  and  Computers  (CC)  network  with 
extraordinary  range,  capacity  and  dependability.  The  CC 
network  provides  secure,  reliable  access  to  information 
sources  over  extended  distances  and  complex  terrain. 

The  CC  network  is  primarily  embedded  within  the 
mobile  cells  and  moves  with  the  other  combat 


vehicles  in  formation  and  can  be  replaced  adhoc  in 
case  of  annihilation,  strengthening  the  capability  of 
the  command,  control,  communications,  computers, 
intelligence,  surveillance,  and  reconnaissance  (C4ISR) 

[3]  network  on  the  move.  The  FCS  CC  communication 
network  consists  of  several  existing  communication 
systems  such  as  Joint  Tactical  Radio  System  (JTRS) 

Clusters  1  [4]  and  5  [5]  with  Wideband  Network  Waveform 
(WNW)  and  Soldier  Radio  Waveform  (SRW),  Network 
Data  Link  and  Warfighter  Information  Network-Tactical 
(WIN-T).  Most  of  these  systems  emerged  from  earlier 
technologies  and  are  not  designed  to  accommodate 
future  FCS  bandwidth  requirements.  Every  FCS  vehicle  in 
the  Unit  of  Action  (UA)  should  be  equipped  with  a  4-  or 
8-channel  JTRS  Cluster  1.  Each  channel  has  constrained 
bandwidth  capabilities.  Soldiers  and  other  weight  and 
power-constrained  cells  should  be  equipped  with  a 
1- or  2-channel  JTRS  Cluster  5.  In  addition  to  the  WNW 
and  SRW  communications  backbone,  the  software 
programmable  JTRS  will  support  other  waveforms  to 
ensure  current  force  Joint,  Interagency  and  Multinational 
(JIM)  interoperability.  The  WIN-T  will  provide  additional 
communications  capability  within  the  UA,  as  well  as 
reach  to  communication  layers  above  —  intra-  and  inter- 
UA,  and  UA  to  Unit  of  Employment  (UE)  — and  range 
extension. 

Multiple  vehicles  and  communication  links  support 
the  operation  and  deployment  of  FCS  vehicles. 

Figure  1  depicts  the  support  vehicles  and  network 
communications  required  to  conduct  a  multi-national 
force.  Since  each  mission  is  different,  the  characteristics  of 
the  mission  will  dictate  the  vehicles  and  communications 
involved.  Nevertheless,  most  missions  will  require 
an  Unmanned  Air  Vehicle  (UAV)  to  provide  real-time 
surveillance  information  regarding  the  terrain,  battle 
conditions  and  enemy  location.  The  Central  Air  Data 
Computer  (CADC)  will  relay  communications  to  satellites 
that  pass  on  information  to  the  C4I  grid. 


FCS  Scenarios  and  Embedded  Training  (ET)  for 
CES  Stimulation 

Embedded  Training  (ET)  capability  will  be  developed 
as  an  integral  part  of  the  FCS  manned  cell  and  C4ISR 
architectures  [6].  Specifically,  while  the  vehicles  are 
transported,  training  exercises  are  executed  as  simulations 
that  mimic  the  actual  engagement  scenario. 
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Therefore,  ET  generates  additional  traffic  that  the 
CES  simulators  should  accommodate  for  realistic  CES 
bandwidth  tests  results.  Knowing  how  much  bandwidth 
the  current  technology  can  accommodate  is  an  important 
metric  reflected  in  future  designs. 

A  typical  FCS  scenario  is  depicted  in  figure  1 .  It 
evolves  different  radios  and  systems  that  control  the 
communication  on  the  move  and  re-establish  connectivity 
in  case  of  attack.  Hence,  the  FCS  communication  network 
to  be  evaluated  for  ET  consists  of  a  group  of  systems 
such  as  JTRS  Clusters  1  and  5  with  WNW,  as  well  as  SRW, 
Network  Data  Link  and  WIN-T. 

The  communications  between  vehicles  of  the  UA  is 
handled  by  the  JTRS  Cluster  1.  Dismounted  solders  or 
weight  sensitive  vehicles  are  equipped  with  a  1- or  2- 
channelJTRS  Cluster  5.  Due  to  secrecy,  the  transmission 
and  wave  characteristics  of  JTRS  radios  are  not  available 
for  publication,  but  are  essential  to  realistically  model 
the  wave  and  transmission  patters  that  interact  with 
any  terrain  database  and  weather  characteristics  of  a 
simulation.  Typical  values  are  provided  by  some  CES 
simulators  but  not  released  for  publication  due  to 
contractual  restrictions. 


Figure  1 :  Depiction  of  FCS  Scenario  [7] 

FCS  Communication  Effects  Simulators 

Three  different  emerging  CES  were  studied,  COMPOSER, 
QUALNET  and  Omnet-H-.The  COMPOSER  provides  models 
that  through  a  composition  tool  can  be  placed  in  a  certain 
location  and  interact  with  United  States  Geological  Survey 
(USGS)  terrain  data.  However,  the  radio  models  are  being 
developed  and  allow  modifications.  Its  network  hierarchy 
is  proprietary  and  allows  no  modification.  Qualnet  is  a 
commercial  tool  made  by  Scalable  NetworkTechnologies 
(SNT).This  simulator  includes  FCS  radio  models  as  part 


of  its  FCS  package.  JTRS  and  WIN-T  models  are  available 
under  unique  license  agreements.  Qualnet  can  interact 
with  terrain  databases  and  provide  FCS  models.  Omnet++ 
is  an  open  source  communications  simulator  that  can  be 
100%  customized  [8].  Interaction  with  3D  data  is  limited, 
but  can  be  improved  through  further  development.  All 
three  CES  provide  a  3-dimensional  visual  image  of  the 
undergoing  communications  and  the  ability  to  preset  cells 
at  a  certain  Global  Positioning  System  (GPS)  location  for 
analysis.  However,  modifications  to  the  communications 
network  initial  layout  are  an  evolving  capability.  In 
contrast,  open  source  OMNET++  provides  source  code 
that  can  be  modified  and  manually  customized  [9]. 
Network  models  are  based  on  pre-configured  models, 
but  each  model  can  be  copied  and  customized  to  model 
different  protocols  and  designs.  Different  examples  model 
wireless  and  LAN  networks  that  can  be  adapted  to  any 
given  model  specifications.  Existing  models  can  represent 
Mobile  Internet,  Mobile  Data  Link  and  802.1 1  among 
others. 

SNT-Qualnet  takes  a  different  approach  providing 
a  unique  simulator  that  structures  the  FCS  network 
using  a  layered  approach  based  on  the  Open  Systems 
Interconnection  Basic  Reference  Model  (OSI)  [10]. The  FCS 
network  architecture  represented  in  SNT  is  a  hierarchically 
structured  network.  The  architecture  consists  of  the  FCS 
proprietary  Mobile  Intranet  (Ml)  Layer  and  the  Mobile  Data 
Link(MDL)  Layer.  The  Ml  Layer  is  mainly  responsible  for 
the  routing  support  while  the  MDL  layer  provides  medium 
access  to  the  wireless  channel  [1 1  ]. 

The  Ml  Layer  includes  the  following  models: 

•  Region  formation 

•  Region  Access  Point  (RAP)  selection 

•  Subnet  formation 

•  Gateway  selection 

•  Intra-Region  routing  using  Hazy  Sighted  Link  State 
Routing  Protocol  (HSLS) 

•  Inter-Region  routing  using  Multi-level  Abstracted 
Link  State  Routing  (MALSR) 

•  Radio  Open  Shortest  Path  First  (ROSPF) 

•  Multi-Point  Relay  (MPR) 

The  MDL  Layer  includes  the  following  model  of  Unified 
Slot  Assignment  Protocol  (USAP).  At  the  lowest  level  of  the 
hierarchy  are  Cells  (or  nodes). 

As  illustrated  in  Figure  2,  all  nodes  in  the  network  are  cells. The 
Cells  dynamically  form  (and  re-form)  individual  Regions  using  a 
Region  formation  algorithm. 
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Subnets  are  predetermined.  A  Subnet  can  have  more  than  one 
Gateway  and  a  Gateway  can  belong  to  more  than  one  Subnet. 
In  fact,  routing  within  a  Region  is  accomplished  using  HSLS.To 
route  between  Regions,  MALSR  is  employed  and  is  executed  on 
the  RAPs. 


For  FCS  environments  such  as  those  depicted  in  Figure 
1,  SNT  provides  a  basis  that  was  adapted  to  FCS  network 
traffic  according  to  current  FCS  specifications. This  includes 
support  for  JTRS  radio  cluster  5  with  two  channels,  thus 
limiting  our  experiments  to  scenarios  based  on  land 
operation  and  inter-vehicle  communication  such  as  the 
example  quoted  earlier  regarding  the  deployment  of  an 
objective  force  to  a  location  near  the  Caspian  Sea. 

Omnet++  and  COMPOSER  CES  simulators  studied  do  not 
provide  a  layered  approach  or  a  path  to  configure  each 
cell  internally.  Most  cells  could  be  configured  as  a  network 
node,  but  the  communications  modeling  fidelity  of  that 
approach  may  not  be  sufficient.  Other  issues  such  as 
mobile  communications  may  not  be  correctly  modeled. 
The  JTRS  radios  models  were  not  available  for  use  in  this 
study;  nonetheless  all  CES  studied  proved  useful  to  identify 
issues  on  Line-of-Sight  (LOS)  communications  on  variable 
altitude  terrain. 


Finally,  routing  between  Subnets  is  achieved  using  ROSPF  on 
the  Gateways.  Medium  Access  Control  (MAC)  within  Subnets  is 
achieved  using  USAP.  Thus,  transmissions  within  and  between 
Regions  of  a  Subnet  rely  on  USAP.  However,  transmissions 
between  Gateways  may  use  any  medium  access  protocols,  such 
as  Carrier  Sense  Multiple  Access  (CSMA),Time  Division  Multiple 
Access  (TDMA),  or  even  USAP.  The  choice  of  which  MAC  protocol 
to  use  between  Gateways  depends  on  the  radio  equipment  the 
Gateways  are  equipped  with.  Figure  3  depicts  HSLS  and  MALSR 
and  how  ROSPF  connects  gateways. 


Location  of  Scenario  Selected  for  CES  Evaluation. 

The  Caspian  Sea  is  the  standard  location  used  for  FCS 
communications  testing  located  between  latitude  48°  N 
and  36°  N  and  longitude  47°  E,  and  54°  E,  covering  more 
than  293  thousand  square  miles. 

Given  the  immensity  of  the  area  we  will  concentrate  on  a 
small  scenario  based  out  of  the  Atyarau  airport  located  at 
47°  07'  27.90"  N  and  51°  49'  29.92"  E.  Figure  4  depicts  the 
location  courtesy  of  Google  Maps. 


Figure  4:  Atyarau  Airport 

The  Atyarau  Airport  was  selected  due  to  its  proximity  to  the 
North  shore  of  the  Caspian  Sea.  However,  any  location  in 
the  world  will  work  for  the  test.  _  _ 
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The  Caspian  Sea  particularly  suits  the  FCS  scenario 
mentioned  earlier  regarding  vehicles  deployed  for  combat. 
The  airport  has  buildings  and  access  roads  easy  to  map  using 
Global  Positioning  System  (GPS)  coordinates.  Moreover, 
the  terrain  elevation  is  easy  for  any  vehicle  to  traverse  and 
is  included  in  the  Caspian  Sea  USGS  database. 

The  possibilities  to  find  Digital  Elevation  Model  (DEM)  and 
Digital  Terrain  Elevation  Data  (DTED)  terrain  databases  are 
high  due  to  its  proximity  to  the  Caspian  Sea,  but  were  not 
verified.  GPS  coordinates  can  be  used  to  position  each 
communication  cell  and  use  a  waypoint  feature  to  move 
each  vehicle  (cell)  across  the  terrain  as  desired,  as  their 
interaction  with  weather  and  terrain  is  observed.  The 
suggested  location  for  the  scenario  is  irrelevant  for  the 
results  of  the  experiment.  Any  location  in  the  globe  will 
be  sufficient  for  research.  Therefore,  any  available  DEM 
or  DTED  data  can  be  used  to  establish  waypoints  for  the 
experiment. 


Network  Components  Selected  for  Simulation 

In  [1 2],  MITRE  suggests  that  the  initial  hardware  to  be  used 
is  mostly  COTS  while  JTRS  radios  connect  the  subnets  and 
nets  together.  Therefore  we  will  concentrate  on  the  use 
of  JTRS  radios  for  the  inter-cell  communication.  Mobile 
Ad  Hoc  Network  (MANET)  routing  software  that  runs 
on  Linux  provides  IEEE  802.11  connectivity  for  which 
wireless  protocols  are  also  included  as  part  of  the  subnet 
communication.  SNT  FCS  version  provides  JTRS  radio 
models  based  on  their  abstract  model  configured  to  JTRS 
specifications,  while  nets  and  subnets  are  routed  using 
FCS  proprietary  Mobile  Intranet  Protocols  (Ml)  which 
define  the  Hierarchy  formation  protocols  and  the  routing 
protocols.  RAP  protocols  define  the  regions  while  the 
subnets  are  defined  by  HSLS.  The  ROSPF  protocol  and  the 
Multi  Relay  Protocol  control  the  routing.  The  Mobile  Data 
Link  Layer  uses  USAP  (Boeing-FCS-USAP).  The  physical 
layer  is  already  defined  for  FCS  based  on  predefined  FCS 
physical  layer  characteristics  based  on  the  antenna's 
radiation  patterns,  frequency,  bandwidth  and  gain. 
Protocol  detail  is  not  provided  as  it  is  beyond  the  scope  of 
this  paper.  However,  it  is  essential  to  understand  the  details 
of  FCS  communication  to  interconnect  available  models. 

Our  proposed  baseline  can  be  used  on  any  FCS  capable 
CES.  We  have  selected  a  straightforward  scenario 
that  can  produce  measurable  results  that  verify  CES 
operation.  The  use  of  JTRS  radios  is  essential  since  they 
carry  the  communication  between  cells.  Other  models 
such  as  MANET,  Ml  and  RAP  provide  low  level  wireless 


communication,  but  may  not  be  available  on  all  simulators. 
Available  COTS  models  such  as  IEEE  802.1 1  can  also  be  used 
to  simulate  wireless  networks.The  idea  is  test  the  interaction 
of  the  available  models  on  the  CES  with  the  terrain  from 
the  database.  Since  FCS  communications  is  loose  and  not 
yet  defined  and  constantly  evolving,  models  provided 
from  each  simulator  will  vary,  but  their  performance  can  be 
measured  if  a  baseline  scenario  is  used. 


Experiments 

All  three  simulators  were  used  for  this  test,  but  only  SNT 
will  provide  a  release  for  this  publication.  Confidentiality 
agreements  make  impossible  the  publication  of  the  other 
results.  Nevertheless,  our  interest  is  to  provide  a  testing 
model,  not  a  review  of  ongoing  CES  development. 

Our  proposed  baseline  consists  of  two  groups  of  vehicles 
in  proximity  and  moving  separately  but  supporting  each 
other  so  that  the  communication  failures  can  be  observed 
as  vehicles  move  out  of  any  communications  range  or  as 
transmission  is  obstructed  by  terrain.  Each  group  consists  of 
a  subnet  of  five  vehicles  that  share  a  network  with  another 
group  of  five  vehicles  that  form  their  own  subnet. 

As  specified  by  FCS  standards,  each  cell  that  moves  through 
the  terrain  and  experiences  loss  of  communication  due  to 
terrain  obstacles,  distance  or  destruction  by  the  enemy 
should  regroup  and  reestablish  new  connections  with 
unobstructed  or  existing  cells.  Therefore,  it  is  important  to 
establish  a  communications  baseline  between  the  studied 
cells  to  checkthat  cell  behavior  is  correct.  Initial  tests  are  used 
as  the  communications  measurement  best  case  scenario. 
Hence,  the  initialization  of  the  network  can  be  monitored 
and  checked  for  typical  FCS  operation.  Once  a  baseline 
has  been  established,  the  mobility  of  the  vehicles  can  be 
programmed  to  follow  the  streets  near  the  airport  using  GPS 
waypoints.  Observations  can  then  be  made  to  determine 
JTRS  radio  limitations  when  compared  to  the  model. 

SNT  CES  provides  an  FCS  infrastructure  to  enable  the  user  to 
experiment  with  the  location  and  movement  of  FCS  vehicles 
considering  different  protocols  limited  by  the  capacity 
of  the  JTRS  radios.  Other  CES  studied  were  not  as  flexible, 
but  internal  file  modification  permitted  relocation  and  GPS 
waypoint  following. 

We  configured  SNT  with  the  proposed  scenario,  where 
both  subnets  (cells)  were  separated  approximately  1.5  Km, 
and  without  using  a  terrain  database.  Initially  simulated 
experimental  results  show  that  the  network  has 
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no  connectivity  for  about  twenty  seconds,  which  is  the 
approximate  time  that  the  network  needs  to  initialize  its 
mobile  net  and  subnets. 

Examination  of  the  physical  layer  presents  a  solid 
transmission  of  signals  throughout  the  200  minutes  of 
simulation,  proof  of  the  nearby  location  of  the  other  cells 
and  the  lack  of  interference  from  the  terrain. 

On  the  graphs  shown  the  x-axis  corresponds  to  the  number 
of  nodes  in  the  simulation  where  two  cells  of  5  nodes  each  are 
tested.  Cell  0  includes  nodes  1,  2,  3, 4  and  5.  Similarly,  cell  1 
contains  nodes  6, 7, 8, 9  and  10.  Pair  Nodes  1  and  5,  and  7  and 
10  are  connected  directly  through  Ethernet.  Each  cell  moves 
through  the  designated  area  increasingly  away  from  each 
other  for  200  minutes.The  colors  of  the  bars  represent  each  cell 
group  plotted,  blue  for  cell  0  and  green  for  cell  1 . 

In  figures  5  and  6,  most  signals  get  passed  to  the  MAC  layer 
with  higher  concentrations  on  cells  3  and  7  due  to  proximity 
to  each  other. 


Figure  5:  Abstract  Signals  Transmitted 
Figure  6:  Abstract  Signals  Detected 


Observe  that  not  all  the  signals  transmitted  by  a  source  at 
the  physical  layer  were  received  at  their  destinations  due 
to  inherent  losses  during  transmission. 

On  the  following  OSI  layer,  MAC  [1 3][1 4]  the  packets 
received  from  the  network  layer  have  no  Unicast  packets 
component  on  the  network,  which  agrees  with  High  Level 
Architecture  (HLA)  using  only  UDP  and  Broadcasting.  A 
description  of  HLA  will  exceed  the  contents  and  scope  of 
this  paper,  but  in  general  terms  HLA  only  broadcasts  to 
members  of  its  network  using  UDP.  For  more  details  on 
HLA,  see  [15].  Signals  received  and  forwarded  to  the  MAC 
layer  are  less  than  the  abstract  signal  transmitted  initially  as 
expected.  In  figure  7,  notice  how  nodes  4  and  5  experience 
some  communication  loss  for  this  test  due  to  HLA  nature. 


Not  all  packets  transmitted  are  used  by  HLA,  only  UDP.  In 
figure  8,  most  broadcast  packets  are  received  from  the 
network  layer  due  to  HLA's  broadcast-only  nature. 


Figure  7:  Abstract  Signals  Received  and  Forwarded  to  MAC 
Figure  8:  USAP  Broadcast  Packets  Received  from  Network  Layer 


On  the  transport  layer  no  UDP  and  TCP  packets  are 
observed  [1 3][1 4],  only  UDP  packets  as  the  H  LA/D  IS 
protocol  specifies.  As  seen  in  figure  9  the  activity  of  UDP 
packets  is  almost  constant  for  each  node;  there  is  no  TCP 
activity  as  expected  in  HLA. 


Figure  9:  UDP  Packets  from  Application  Layer 

Observations  can  be  made  after  the  baseline  is  executed  in 
each  CSE  simulation.  The  user  should  lookfor  discrepancies 
regarding  the  communication  channel  characteristics  and 
its  interaction  with  its  surroundings  and  terrain.  Note  that 
different  signals  at  different  frequencies  travel  different 
paths  [16].  A  JTRS  signal  may  overcome  obstacles  while 
operating  at  lower  frequencies  otherwise  impossible  to 
overcome  at  Ultra  High  frequencies.  Iteratively,  the  fidelity 
of  each  CES  can  then  be  verified  as  well  as  its  embedded 
training  capabilities. 
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Figure  10  depicts  our  test  simulation  performed  on  the 
Qualnet  SNT  simulator  at  the  Atyarau  location  with  the 
cells  in  the  strategic  initial  GPS  locations.  In  this  simulation 
the  connectivity  between  all  the  cells  was  evaluated  to 
determine  that  the  behavior  of  random  communication 
between  the  cells  is  acceptable  within  the  expectations 
of  the  data  transport  and  the  movement  of  the  cells.  The 
graphs  in  figures  6  to  9  show  that  an  initial  transmission  or 
random  signal  loses  data  as  it  traverses  the  different  layers 
of  the  OSI  model  due  to  HLA  operating  with  a  subset  of  the 
initial  data  concentrated  on  UDP  packets.  Nodes  4  and  5 
suffer  from  below  average  communication  since  they  are 
the  cells  that  moved  the  furthest  from  each  other. 


Figure  1 0:  FCS  Test  Scenario  in  Qualnet 


Nevertheless,  setting  up  a  simple  scenario  for  testing 
FCS  mobility  as  suggested  is  rapid  if  the  models  behave 
as  expected.  Given  that  the  CES  in  question  provides  the 
flexibility  to  set  initial  conditions  such  as  terrain  location 
and  cell  configuration,  a  small  scenario  can  be  built  to  verify 
the  CES  accuracy  before  committing  to  a  larger  exercise. 

For  this  particular  CES  test,  the  initial  GPS  location  for  each 
cell  was  defined.  Using  random  paths  is  not  recommended 
since  it  leads  to  the  inability  to  establish  a  baseline  for 
comparison  between  different  CES  tools. 

After  setting  the  paths  for  each  cell,  considerations  about 
the  weather  can  be  added.  Setting  up  the  correct  path 
is  important  since  the  distances  between  the  cells  and 
interactions  with  other  cells  affects  the  communication. 
Cells  traversing  between  many  obstructions  can  lose 
their  capability  to  communicate.  Distances  over  5  Km  for 
handheld  radios  and  15  Km  for  Small  Form  Fit  (SFF)  radios 
are  common  with  LOS  free  of  interruptions.  With  radio 
bandwidth  limited  to  2  Mbps,  real-time  video  is  impossible 
without  interruption.  Therefore  selecting  a  difficult  path 


and  initial  long  distances  between  the  cell-vehicles  is 
not  recommended,  but  can  determine  the  extent  of  the 
communications  as  the  cells  try  to  regroup  their  subnet  as 
part  of  a  different  region.  Likewise,  embedded  training  in 
addition  to  the  subnet  traffic  may  decrease  the  ability  to 
communicate  correctly  by  limiting  the  available  bandwidth. 
Some  simulators  provide  an  opportunity  to  embed 
additional  traffic  in  parallel  such  as  the  SNT  simulator 
presented  using  distributed  CES  which  interacts  with 
the  Message  Transceiver  System  (MTS).  MTS  is  a  message 
system  that  allows  the  CES  to  interact  with  other  parts  of 
the  System  of  Systems  Virtual  Framework  (SVF). 

Figure  11  depicts  our  suggested  baseline  for  Embedded 
Training  FCS  communications  testing.  The  scenario 
consists  of  two  subnets.  This  can  be  increased  to  4  subnets 
if  computation  power  is  not  a  constraint.  Each  subnet 
consists  of  five  mobile  FCS  communication  cells  with  a 
starting  distance  between  them  no  larger  than  10  Km 
which  is  within  the  limit  of  JTRS  communication.  All  cells 
within  each  subnet  will  initiate  travel  together  and  there 
should  not  be  further  than  0.5  kilometers  between  cells  to 
accommodate  minimum  transmission  requirements  for  all 
radios.  These  form  a  region  after  the  initial  20  sec  network 
setup  time. 
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Figure  1 1 :  Baseline  Scenario 
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Within  the  next  10  minutes  of  travel  each  cell  will  disperse 
from  the  others  in  a  star  configuration  at  a  speed  (in  KPH) 
no  faster  than  Vi  the  total  distance  in  Kilometers  of  the 
exercise  to  allow  1 20  minutes  of  simulation  time.  This 
scenario  is  used  to  place  demands  on  the  FCS  routing  to 
regroup  cells  into  other  regions  that  connect  with  cells  in 
proximity.  Meanwhile  the  opposite  subnets  will  be  moving 
towards  the  airport  trying  to  establish  communication 
if  it  is  not  yet  already  established.  After  20  minutes  of 
simulation  time  the  cells  will  start  regrouping,  but  following 
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the  paths  or  roads  assigned  by  preprogrammed  waypoints 
at  a  constant  speed  (in  KPH)  no  faster  than  Vi  the  total 
distance  in  Kilometers.  During  this  period,  the  signal  and 
communications  should  strengthen,  and  separate  regions 
for  a  subnet  become  a  single  region  again.  From  this  point 
onward  as  the  two  (or  four  subnets)  get  in  proximity,  new 
regions  will  be  formed. 

The  bandwidth  will  shrink  as  all  packets  on  the  network 
intensify  their  collision  rate  with  increasing  obstacles  and 
interference  effects  [13][14].  The  overall  communications 
of  the  small  exercise  presents  a  normal  behavior  for  LOS 
communication  between  the  cells.  The  interaction  with 
terrain  database  should  show  a  different  behavior  that  we 
are  unable  to  test  here  due  to  limitations  on  terrain  data 
access. 

Similar  evaluations  can  be  done  to  verify  the  accuracy  of 
any  communications  simulator  rapidly.  The  setup  time  is 
less  than  30  minutes  and  the  execution  time  less  than  30 
minutes. 


of  the  CES  evaluated  had  the  capability  to  change  model 
behavior  as  their  transmission  frequency  was  changed. 
Remember  that  waves  travel  differently  and  diminish  their 
ability  to  travel  through  solids  as  their  frequency  increases. 

Not  all  CES  will  produce  the  same  output  results.  By 
performing  the  suggested  Black  Box  testing,  a  heuristic 
observation  of  the  operation  can  be  sufficient  to  determine 
the  capability  of  the  product  for  this  purpose.  However,  it  is 
more  daunting  to  quantify  the  results  when  the  details  of 
these  designs  are  classified  or  difficult  to  locate.  Therefore 
a  behavioral  analysis  is  more  convenient  and  a  preamble  to 
detailed  results. 

Access  to  military  terrain  data  can  be  difficult,  but 
interaction  between  terrain  and  communications  can  be 
done  with  available  open  source  data.  Most  of  the  CES 
evaluated  had  the  ability  to  read  DTED  and  USGS  data  as 
well  as  other  formats. 


Future  Work 


Conclusion 

The  estimation  of  the  capabilities  of  a  particular  CES  is  no 
easy  task.  The  evaluation  of  its  usability  and  fidelity  can 
be  done  rapidly  with  small  scenarios  that  evaluate  known 
network  properties  of  the  models  used  by  the  simulator  in 
a  black  box  test.  By  using  a  priori  analysis  of  the  models 
compared  with  the  original  design  characteristics,  it  is 
possible  to  explore  the  limits  of  JTRS  radios,  for  example, 
and  interconnected  networks  and  verify  the  simulator's 
fidelity. 

The  use  of  JTRS  radios  as  the  main  hubs  for  WIN-T  networks 
on  a  moving  battalion  [17]  communication  is  a  limiting 
factor  in  the  success  of  embedded  training  in  situations 
where  video  information  is  needed  for  enemy  awareness 
especially  in  broadcast  HLA  format.  CES  experiments 
can  verify  and  suggest  the  best  configuration  for  ideal 
bandwidth  optimization. 

Testing  JTRS  capabilities  and  limitations  can  be  done  with 
existing  CES  by  exploring  their  capabilities  on  a  small 
controllable  network  where  the  interactions  with  terrain 
and  the  distances  needed  to  communicate  are  controlled. 
By  setting  a  small  group  of  vehicles  as  cells  as  suggested, 
bandwidth  studies  are  possible.  Moreover,  similar 
experiments  can  verify  the  fidelity  of  the  radiation  pattern 
of  each  radio  as  it  travels  through  terrain  and  check  its 
capabilities  at  different  frequencies.  Please  note  that  none 


Ultimately  we  will  like  to  have  access  to  DTED  or  USGS  DEM 
Caspian  Sea  data  for  the  airport  presented  in  this  paper 
to  test  the  nuances  of  the  CES  simulators.  We  will  like  to 
run  the  same  scenario  and  observe  variations  in  data  as 
the  vehicles  traverse  through  obstacles  such  as  walls  and 
mountains.  Moreover,  some  of  these  products  advertise 
the  ability  to  interact  with  weather,  a  capability  we  would 
like  to  compare  with  terrain  and  non-terrain  interaction 
and  postulate  a  better  model  to  verify  the  fidelity  of  each 
particular  CES  module. 

As  new  technology  emerges  and  the  SOSCOE 
implementation  is  defined,  other  CES  products  will  emerge. 
Users  will  need  a  baseline  to  test  the  initial  fidelity  of  the 
tool  and  compare  bandwidth  results  with  other  tests  to 
ensure  fidelity.  Our  proposed  baseline  can  be  used  to  assist 
users  in  obtaining  a  certain  degree  of  confidence  about  the 
fidelity  of  the  CES  tool  built,  evaluated,  or  purchased 

Other  commercial  simulators  such  as  OPNET  can  be  used 
with  our  proposed  scenario  to  observe  their  capabilities 
and  determine  their  suitability  for  FCS  terrain  and  weather 
effects  simulation. 
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Acronyms 


References 


C4I  Command,  Control,  Communications,  Computers, 
Intelligence 

C4ISR  Command,  Control,  Communications,  Computers, 
Intelligence  and  Reconnaissance. 

CADC  Central  Air  Data  Computer 
CC  Communications  and  Computers 

CES  Communication  Effects  Server 

COTS  Commercial  Off  The  Shelf 
CSMA  Carrier  Sense  Multiple  Access. 

DEM  Digital  Elevation  Model 

DTED  Digital  Terrain  Elevation  Data 

ET  Embedded  Training 

FCS  Future  Combat  Systems 

FoS  Family  of  Systems 

GPS  Global  Positioning  System 

HLA  High  Level  Architecture 

HSLS  Hazy-Sighted  Link  State  Routing  Protocol 

IEEE  Institute  of  Electrical  and  Electronical  Engineers 

JIM  Joint  Interagency  Multinational 

JTRS  Joint  Tactical  Radio  Systems 

LAN  Local  Access  Network 

LOS  Line  of  Sight 

MAC  Medium  Access  Control 

MALSR  Multi-level  Abstracted  Link  State  Routing 

MANET  Mobile  Ad  Hoc  Network 

MDL  Mobile  Data  Link 

Ml  FCS  proprietary  Mobile  Internet 

MPR  Multi-Point  Relay 

MTS  Message  Transmitter  System 

OF  Objective  Force 

OSI  Open  Systems  Interconnection  Basic  Reference 

Model 

RAP  Region  Access  Point 

ROSPF  Radio  Open  Shortest  Path  First 

SFF  Small  Form  Fit 

SNT  Scalable  NetworkTechnologies 
SOSCOE  Systems-of-Systems  Common  Operating 
Environment 

SRW  Soldier  Radio  Waveform 

SVF  Systems  of  Systems  Virtual  Framework 

TDMA  Time  Division  Multiple  Access 

UA  Unit  of  Action 

UAV  Unmanned  Air  Vehicle 

UE  Unit  of  Employment 

USAP  United  Slot  Assignment  Protocol 

USGS  United  States  Geological  Survey 

WIN-T  Warfighter  Information  Network -Tactical 

WNW  Wideband  Network  Waveform 
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ABSTRACT :  Highly  advanced  sensor  technologies  give 
our  military  commanders  a  significant  command  and 
control  (C2)  advantage  over  our  enemies  during  conflicts ; 
particularly  with  respect  to  situation  awareness  (SA).  The 
use  of  advanced  sensor  technology  models  in  synthetic 
battlespace  gives  war  fighters  parallel  advantages.  Two 
accepted  simulation  methodologies  for  analyzing  the  impact 
of  sensor  technologies  are  through  Human-in-the-Loop 
(HITL)  experiments,  such  as  Joint  Urban  Operations  (JUO), 
which  utilize  sensor  capabilities  to  assist  human  participants 
during  the  experiments ,  and  Monte  Carlo  constructive  (MCC) 
simulations ,  which  can  be  used  to  model  human  performance. 
In  HITL  experiments  using  Joint  Semi-Automated  Forces  (JSAF), 
participants  describe  their  SA  using  Situation  Awareness 
Objects  (SAOs)  which  then  can  be  reconstructed  using 
Endsley's  (1995)  three  levels  ofSA  ( perception ,  comprehension , 
and  prediction).  MCC  experiments ;  which  are  dominated  by 
algorithmically  determined  behaviors ;  can  be  used  to  model 
SA.  Sensor  measurements  currently  can  be  fused  to  perceive 
individual  entities ;  but  do  not  have  the  capability  to  recognize 
groupings  of  entities,  resulting  only  in  partial  perceptual 
SA.  Furthermore,  current  sensor  data  fusion  models  do  not 
produce  the  second  and  third  levels  ofSA,  comprehension  and 
prediction. 

This  paper  will  report  research  efforts  to  utilize  both 
methodologies  to  expand  the  use  of  SAOs  beyond  player 
declarations  to  the  automatic  generation  of  SAOs.  We  develop 
a  method  to  organize  events  drawn  from  scenarios  taken  from 
HITL  experiments  using  SAOs  in  order  to  develop  situation 
awareness  algorithms  for  the  MCC  runs.  These  model- 
generated  synthetic  SAOs  (SSAOs)  can  be  compared  to  SAOs 
generated  by  human  players  to  identify  the  accuracy  of  the 
models  as  well  as  be  used  to  identify  strengths  and  weaknesses 
in  player  performance. 


Introduction 

This  paper  focuses  on  building  a  foundation  for  a  research 
effort  on  modeling  situation  awareness  (SA)  in  synthetic 
theater  of  war  (STOW).  We  present  a  relevant  research 
problem,  and  a  description  of  how  it  can  be  modeled. 

We  focus  on  SA  because  it  has  widespread  relevancy 
throughout  the  military  community  and  at  all  levels  of  the 
command  hierarchy.  We  are  also  interested  in  SA  because, 
as  a  complex  mental  state  that  is  reflects  numerous 
cognitive  processes,  it  is  a  particularly  challenging 
modeling  problem.  Being  able  to  successfully  model  SA 


will  have  at  least  a  two-pronged  benefit,  in  our  view.  First, 
it  would  validate  our  assumptions  of  the  results  of  Human- 
in-the  Loop  (HITL)  exercises  in  which  human  participants 
are  a  part  of  the  simulation.  Second,  our  approach  is 
applicable  to  SA-related  issues,  including  command  and 
control  (C2)  and  training. 


Motivation 

There  are  two  accepted  experimental  methods  for 
evaluating  sensors  technology:  HITL  experimentation  and 
Monte  Carlo  Constructive  (MCC)  experiments,  which  are 
statistic-based  constructive  experimentation  of  sensor 
models.  For  HITL  experiments,  SA  output  is  a  function  of 
human  behavior.  The  use  of  constructive  runs,  up  to  this 
point,  in  sensor  modeling  and  simulation  experiments 
has  been  conducted  independently  of  consideration 
for  human  interactions  and  attempts  to  model  situation 
awareness  have  been  limited  to  its  more  perceptual 
aspects.  HITL  experiments  yield  a  wealth  of  data  and  if  the 
MCC  methodology  can  be  used  to  develop  tools  to  give 
analysts  a  synthesized  encapsulation  of  events  akin  to  the 
information  provided  by  the  HITL  players,  they  can  make 
better  use  of  resources  (time  and  personnel)  to  make 
better  decisions. 

Our  approach  is  directly  tied  to  ongoing  HITL 
experimentation  by  the  Forces  and  Modeling  Simulation 
(FMS)  Group  in  the  J9  Directorate  at  the  US  Joint  Forces 
Command  (JFCOM),  the  evolving  sensor  modeling 
technology  byToyon  Research  Corporation,  and  the 
research  and  support  in  synthetic  battlespace  by  Alion 
Science  and  Technology. 


General  Overview 

Our  proposed  SA  analysis  framework,  in  the  context  of 
STOW,  specifically  in  Joint  Semi-Automated  Forces  (JSAF) 
simulation  software,  can  be  summarized  as  follows.  The 
experiment  consists  of  a  game  that  is  played  among  two 
or  more  potentially  adversarial  forces  (i.e.,  blue,  red,  and 
green  cells).  The  objective  of  the  red  and  blue  cells  is  to 
tactically  outmaneuver  the  adversary.  These  experiments 
operate  on  the  assumption  that  complete  and  superior 
SA,  relying  on  the  aid  of  sensor  technology,  is  the  key  to 
success.  In  these  experiments,  players  demonstrate  SA 
when  they  detect  and  accurately  interpret  sensor  data. 
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For  our  purposes,  Figure  1  illustrates  the  flow  of 
information  through  a  system,  from  sensor  data  (input) 
to  SA  interpretations  (output).  Sensor  data  and  other 
simulation  information  are  fed  into  a  cognitive  "black 
box"  resulting  in  SA.  For  Figure  1  the  signature  refers  to 
an  entity's  identifying  characteristics,  SA  level  2  includes 
the  recognition  of  behavioral  patterns  and  kinematics 
(motion)  and  SA  level  3  includes  the  application  of  learned 
behaviors  to  predict  intent. 

sensor  data 

i 


cognitive 

processes 


i 

SA  Level  1  (signature) 

SA  Level  2  (+  pattern  recognition,  kinematics) 

SA  Level  3  (+learning  behavior) 

Figure  1 .  Information  Flow  from  Sensor  to  SA  output 

In  the  case  of  the  HITL  experiments,  the  black  box 
represents  the  human  players' cognitive  processes 
involved  in  updating  their  mental  representation  of  the 
situation  based  on  information  from  sensor  data,  prior 
knowledge,  and  their  previous  SA.The  outputs  of  these 
processes  are  the  SA  products.  In  the  case  of  constructive 
simulations,  as  there  are  no  human  interactions,  the 
formulation  of  these  processes  is  algorithmic. 


Problem  Description 

The  research  problem  considered  here  focuses  on 
developing  a  situation  template 
of  emplacement  to  be  used  for  future  MCC 
experimentation.  We  first  define  situation  awareness  and 
its  use  in  HITL  experimentation.  We  then  describe  the 
sensor  simulation  platform  and  software  and  the  possible 
way  in  which  the  algorithms  are  implemented.  Finally,  we 
will  consider  a  case  study  of  SA  using  data  collected  from 
human  players. 


Situation  Awareness  Defined 

There  tends  to  be  widespread  agreement  as  to  when  good 
and  poor  SA  is  observed,  but  the  numerous  definitions  of 


SA  illustrate  the  difficulty  of  precisely  defining  SA.  Many  of 
these  definitions  are  not  useful  for  our  purposes  because 
they  do  not  provide  a  sufficient  framework  for  specifying 
the  variables  that  are  likely  to  influence  SA.  Perhaps  the 
most  widely  accepted  view  is  that  of  Endsley's  (1 998)  multi¬ 
level  approach.  This  view  has  come  to  be  adopted  by  the 
military  community  for  research,  training,  operations,  and 
other  purposes,  and  provides  a  framework  suitable  for  our 
purposes. 

According  to  Endsley,  SA  can  be  described  as  consisting  of 
perception,  comprehension,  and  projection  (see  Figure  2). 
These  levels  represent  the  products  of  separate  cognitive 
processes,  yet  the  products  from  one  level  are  influenced  by 
those  of  other  levels. 

The  perceptual  level  involves  the  detection,  recognition,  and 
identification  of  elements  that  define  a  specific  situation. 
Perceptual  SA  relies  on  available  sensory  information,  (e.g., 
from  sensors  in  the  case  of  a  player  in  a  HITL  experiment) 
and  the  player's  prior  knowledge  (e.g.,  object  patterns/ 
schemas  activated  in  memory)  to  identify  individual  situation 
elements  and  object  groups  and  their  characteristics.  Level 
1 A  perception  corresponds  to  the  identification  of  individual 
entities  (e.g.,  a  tank);  Level  1 B  perception  corresponds  to  the 
identification  of  a  grouping  of  entities  (e.g.,  a  mechanized 
brigade).  The  sensor  fusion  processes  that  are  involved 
in  associating  tracks  from  different  sensor  sources  or  in 
grouping  entities  reflect  perception. 

The  comprehension  level  (Level  2)  reflects  an 
understanding  of  the  current  situation,  mapping 
perception  to  function.  In  battlespace,  comprehension 
involves  identifying  the  enemy's  current  activities.  Finally, 
the  projection  level  (Level  3)  reflects  predictions  about  the 
trajectory  of  the  situation  based  on  the  products  of  the 
lower  levels  of  SA  and  prior  knowledge.  In  battlespace, 
projection  corresponds  to  intent:  what  will  the  enemy 
do?  Our  contribution  to  this  paper  focuses  on  how 
MCC  experimentation  can  accomplish  the  mapping  of 
perception  to  function  thus  involving  both  Level  1  and  2  SA. 


HITL  Experiments 

The  US  Joint  Forces  Command  (USJFCOM)  conducts  Joint 
Urban  Operation  (JUO)  series  of  exercises  in  synthetic 
domains  using  human-directed  computer  simulation 
tools,  such  as  Joint  Semi-Automated  Forces  (JSAF),  to 
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explore  and  analyze  current  and  future  Joint  war-fighting 
capabilities.  HITL  actions  and  interactions  are  important 
components  of  these  experiments,  where  humans  control 
the  activity  and  influence  the  outcome  of  the  exercises. 
Humans  control  simulated  Intelligence,  Surveillance,  and 
Reconnaissance  (ISR)  sensors  and  use  Situation  Awareness 
Objects  (SAOs)  to  declare  and  share  their  perceptions 
regarding  model  generated  detections  and  track  objects. 


Situation  Awareness  Objects  (SAOs)  in  HITL 

In  order  to  evaluate  the  effectiveness  of  game-play  in  the  JUO 
exercises,  we  employ  a  novel  tool  called  SAO,  which  is  a  method 
of  recording  information  about  red  force  entities  that  has  only 
been  used  this  series  of  experiments  (Anhalt,  2006).The  SAO  is  a 
compact  package  of  information  that  players  create  and  place  on 
a  shared  terrain  map  that  contains  their  thoughts,  assumptions, 
and  understanding  about  the  enemy.  Figure  3  is  an  example 
input  screen  in  JSAF  that  allows  puckers  to  annotate  their  SA  state 
(create  an  SAO)  during  game  play.  SAOs  are  created  by  having 
players  input  information  about  the  state  of  the  entities. 


Collect  Data  from  JUO/HITL  Experiments 

The  data  to  be  used  for  the  model  comes  from  JUO/HITL 
experiments,  including  electronic  data  that  are  captured  and 
archived  during  each  trial  (e.g.,  ground-truth  unit  information  of 
all  enemy,  friendly  and  neutral  entities,  enemy  unit  track  locations 
as  perceived  by  the  sensors  and  players)  and  Situation  Awareness 
Object  (SAO)  information.  Situational  Awareness  Objects  are 
command  and  control  tools  designed  for  players  to  assess  sensor 


results  and  share  their  findings.  SAOs  are  key  to  evaluating  the 
player's  understanding  of  the  battlespace  and  include  player 
comments. 


Sensor  Modeling:  Urban  Resolve  2015 

The  vast  developments  in  the  fields  of  computer 
engineering  and  computer  science  have  allowed  for 
the  efficient  modeling  of  increasingly  complex  and 
computationally  expensive  sensor  systems.  As  technical 
advances  are  made,  additional  resources  become 
available  for  many  problems  that  may  have  strained 
computing  resources  in  the  past,  if  they  were  possible  at 
all.  One  highly  effective  use  of  modeling  and  simulation 
is  the  rapid  prototyping  of  future  systems.  This  use  has 
allowed  researchers  to  implement  and  discover  new  ideas 
based  on  state-of-the-art  technological  advances,  as  well 
as  adapting  to  changing  environments  and  current  day 
military  defense  and  defeat  needs. 

For  the  last  several  years,  JFCOM  HITL  experiments 
have  focused  on  asymmetric  threats  and  have  explored 
advanced  future  sensor  technologies  as  solutions  to 
defeat  these  threats.  In  so  doing,  a  paradigm  shift  has 
occurred  whereby  HITL  player  involvement  was  expanded 
to  involve  interpreting  formerly  incidental  pieces  of 
information,  or  otherwise  insignificant  simulation  artifacts, 
and  recognizing  that  those  events  play  a  formal  part  of 
understanding  the  enemy. 

For  example,  in  Millennium  Challenge  02  and  Urban 
Resolve  Phase  1  in  2004,  only  the  graphical  representation 
of  an  entity  was  relevant;  events  such  as  digging  and 


Figure  2.  Endsley's  (1998)  Multi-Level  Approach  to  Situation  Awareness 
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Figure  3.  An  Example  Input  Screen  in  JSAF 


loitering  within  a  group  were  not  significant  or  were 
simply  not  a  capability  that  existed  in  the  simulation.  As 
implemented  in  the  J9  Directorate's  Urban  Resolve  201 5 
(UR201 5)  experiment  of  2006,  interpreting  these  events 
was  critical  to  understanding  SA.  HITL  players  were  trained 
to  expand  the  scope  of  SAO  involvement  as  the  primary 
means  of  capturing  the  new  pieces  of  information.  SAOs 
became  the  central  component  to  threat  identification 
and  interdiction  within  the  experiment. 


Description  of  SSAO 

This  paradigm  shift  also  had  the  affect  of  creating  a 
larger  separation  between  HITL  and  MCC  results  by 
enhancing  the  overall  impact  human  interpretation  of 
events  had  on  the  experiment  outcome.  With  MCC  based 
experimentation  focused  on  Level  1  SA,  our  contribution 
in  this  paper  will  make  an  evaluation  of  official  UR201 5 
trial  run  SAO  data,  and  use  that  data  as  a  means  to 
facilitate  generating  higher-level  SA  in  MCC  runs. 


Our  ultimate  intent  is  to  successfully  bring  together  the 
most  beneficial  elements  of  HITL  experiments,  namely 
the  unique  perception  abilities  brought  by  players,  and 
the  scalability  and  efficiency  of  an  MCC  experiment. 

To  effectively  model  patterns  of  player  performance, 
the  concept  was  developed  to  automatically  generate 
Synthetic  Situation  Awareness  Objects  (SSAOs)  for 
MCC  experiments.  The  SSAO  is  a  generic  construct  that 
will  facilitate  capturing  all  three  levels  of  SA  in  an  MCC 
experiment  in  a  manner  parallel  to  the  SAO  in  a  HITL 
experiment.  These  objects  encapsulate  and  automate 
the  (1)  detection  of  entities  and  the  grouping  of  these 
detections,  (2)  identification  of  the  activities  of  these 
entities,  and  (3)  derivation  of  heuristic  models  for  intent  of 
the  opposing  force  as  represented  by  the  entities.  Figure  4 
below  demonstrates  the  three  levels  of  SA  that  would  be 
required  to  be  encapsulated  by  an  SAO/SSAO  during  HITL 
and  MCC  Experimentation. 
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Level  1  (S  A)  (perception)  - 

“What  is  Red?  ” 

Level 2  (SA) (com prehension)- 

“What  is  Red  doing?” 

Level  3  (SA)  (projection)  - 

“What  is  Red  going  to  do?” 


Figure  4.  Three  Levels  of  SA  to  be  Captured  by  SAO/SSAO 


SLAMEM:The  MCC  Simulation  Testbed 

JFCOM  sponsored  large  scale  HITL  experimentation, 
including  UR2015,  has  used  the  Simulation  of  the  Location 
and  Attack  of  Mobile  Enemy  Missiles  (SLAMEM™)  for 
simulating  ISR  capabilities  in  the  JSAF  federation.  SLAMEM 
is  an  entity  level,  event  based  simulation  that  was 
developed  for  analyzing  the  performance  of  coordinated 
command,  control,  communications,  intelligence, 
surveillance,  reconnaissance  (C4ISR)  and  targeting  systems 
against  time-critical  mobile  targets.  SLAMEM  has  also 
been  utilized  in  performing  numerous  MCC  experiments 
on  behalf  of  JFCOM.  SLAMEM's  role  in  supporting 
surveillance  and  targeting  activities  includes  analyzing 
advanced  C4ISR  architectures.  SLAMEM  analyses  have 
several  objectives,  including:  (1)  quantifying  the  potential 
improvements  in  effectiveness  provided  by  the  advanced 
architecture;  (2)  deriving  the  performance  required 
from  the  technologies  to  achieve  specific  mission-level 
goals;  and  (3)  developing  new  CONOPS  for  using  the 
technologies  most  effectively.  As  the  threat  environment 
evolves,  it  has  become  more  important  to  consider  human 
perception  factors  when  making  the  above  3  assessments. 


MCC  Experiments 

Monte  Carlo-based  simulations  are  closed  form 
constructive  processes  that  have  no  human  interaction 
during  runtime.  The  results  of  MCC  experiments  are 
dependent  on  the  scenario  metrics  and  random  statistical 
variations  from  run  to  run,  and  are  initiated  with  a  unique 
random  seed.  These  statistical  variations  can  hinder 
meaningful  results  from  MCC  experiments  if  care  is  not 
taken  to  make  sure  a  sufficient  number  of  trial  runs  are 
completed  (that  is,  random  variations  alone  should  not 
dictate  the  outcome  of  any  run).  This,  however,  is  generally 
not  a  road  block  even  when  the  number  of  trial  runs  is 
large.  This  is  due  to  the  fact  that  MCC  runs  do  not  require 
constant  monitoring  and  lend  themselves  rather  well  to 


batch  processing  for  this  reason. 

The  lack  of  the  human  component  allows  for  greater 
scalability  in  the  number  of  variables  experiments  can 
explore. 

A  limitation  of  MCC  runs  is  that  only  the  most  basic 
levels  of  perception  are  considered  for  evaluation. 
Specifically,  for  SA  Level  1 ,  the  acquisition  of  entities 
in  the  environment,  and  ability  to  maintain  persistent 
surveillance  has  been  the  main  focus.  This  is  primarily 
due  to  the  fact  that  there  are  no  human  interactions,  and 
thus  no  human  providing  insight  into  the  problem.  But 
for  MCC  experiments  to  maintain  their  relevance,  they 
must  adapt  to  the  growing  trends  of  enhanced  perception 
requirements. 


Modeling  SA  in  MCC  with  SSAO 

The  HITL  experiments  place  an  emphasis  on  the 
importance  of  human  interactions  and  the  output  of  the 
experimentation  is  a  function  of  human  behavior,  and 
is  measured  using  the  metrics  of  Situation  Awareness 
(Curiel,  Tran,  Anhalt  &  Yao.,  2005).  On  the  other  hand,  up  to 
this  point,  sensors  modeling  and  simulation  experiments 
in  the  context  of  constructive  simulations,  by  definition, 
have  been  conducted  independently  of  consideration  for 
human  interactions.  Notably  missing  is  the  lack  of  focus 
on  a  situation  model. 

Currently,  MCC  experimentation  is  developed  with 
underlying  fusion  algorithms  that  can  provide  a  means 
of  synthesizing  rudimentary  components  of  SA.  These 
components  make  up  the  first  level  of  SA  which  answers 
the  so-called  "what"  question.  In  the  case  of  the  models 
that  have  been  experimented,  the  "what"  question 
answers  the  specific  questions  of  what  is  being  observed 
or  detected  by  the  sensor  models.  They  also  provide, 
with  the  use  of  various  heuristic  algorithms,  the  ability 
to  aggregate  the  detections  into  composite  units,  also 
referred  to  as  Level  1 B  SA  (Tran,  Yao  &  Curiel,  2004).  For 
example,  the  detection  of  a  group  of  "metal"  vehicles  by 
the  sensors  can  be  funneled  through  the  Fusion  Center 
and  the  output  is  classified  as  a  mechanized  brigade 
(Castleberg,  Colon  &  Berger,  2006).  The  ability  to  extend 
the  constructive  experiment  model  to  cover  the  second 
and  third  level  of  SA  in  MCC  experiments  would  provide 
a  more  complete  experimental  framework  that  validates 
the  effectiveness  of  sensor  models  -  and  doing  so  from  a 
statistically  relevant  analysis  standpoint. 
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Automated  Level  1  SA  in  UR2015 

The  UR201 5  HITL  experiment  explored  a  trade  space 
containing  a  wide  array  of  sensor  technologies.  Each 
sensor,  depending  on  its  underlying  phenomenology  as 
well  as  its  quality,  aided  situation  awareness  to  varying 
degrees.  This  variability  was  characterized  by  confusion 
matrices.  Confusion  matrices  were  defined  to  be  unique 
for  each  sensor,  and  also  to  provide  a  perceived  view 
of  the  entities  within  the  environment  based  on  three 
dimensions  (quality,  camouflage,  and  azimuth  angle). 
Confusion  matrices  represent  the  exploitation  processes, 
whether  automated  or  human-aided,  which  transform 
sensor  data  into  detection  and  classification  outcomes. 
The  probability  of  detection,  correct  classification 
and  identification  of  entities  in  the  environment  was 
determined  by  the  following  equation,  commonly  referred 
to  as  Johnson's  criteria  (Johnson,  1 958;  O'Connor,  2003): 

_  (^Mq  +  ^5o) 
n(v-V5o)"-/,vS. 

where 

N  =  the  number  of  resolvable 

cycles  (equivalent  line  pairs)  on  target 
N50  =  the  number  of  line  pairs  on  target 
required  for  P=0.5 
a  =  2.7 
b  =  0.7 

The  outcomes  of  using  Johnson's  criteria  per  entity  were 
used  to  populate  the  values  of  the  confusion  matrices. 
Figure  5  illustrates  a  generic  example  of  a  confusion 
matrix. 
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Figure  5.  Example  of  the  Format  for  a  Confusion  Matrix 


SA  is  initiated  through  sensor  tasking  and  is  developed 
through  outcomes  of  sensor-target  interactions  and 
subsequent  confusion  matrix  draws.  That  is,  if  say  1 0  entities 


fall  into  a  single  beam  of  a  sensor,  each  of  those  10  entities 
would  be  perceived  separately  and  generate  1 0  distinct 
sensor-target  interactions. 

An  important  development  in  the  field  of  modeling  and 
simulation  has  been  the  change  in  focus  from  a  strictly  entity 
based  visual  perception  of  the  enemy,  to  a  more  context- 
based  perception  of  who  is  likely  to  be  an  enemy  (Ceranowicz, 
Torpey,  &  Hines,  2006). This  change  in  methodology  has  had 
a  vast  impact  on  the  M&S  sensor  development  community, 
and  indeed  on  the  players  who  control  the  sensors  and 
interpret  their  output.  The  determination  of  who  is  likely  to 
be  an  enemy  is  no  longer  based  on  what  the  entity  looks 
like,  but  by  viewing  types  of  evidence  such  as  the  behavior 
of  the  entity  at  any  given  time,  the  accessories  carried  by  the 
entity,  and  any  actions  the  entity  happens  to  be  engaged  in. 
The  challenge  of  modeling  and  simulation  is  to  make  sure 
that  each  piece  of  evidence  is  sufficiently  well  modeled  so 
that  a  HITL  player  has  a  chance  to  recognize  the  evidence, 
and  discriminate  with  enough  confidence  targets  of  interest 
amongst  the  larger  general  population. 

The  scope  of  UR201 5  was  defined  to  provide  a  solution  of 
persistent  surveillance  unmanned  aerial  vehicles  (UAVs) 
fitted  with  high  resolution  imagery  and  video  capable 
of  detecting  on  the  highest  zoom  setting  enough  of 
the  pieces  of  evidence  to  address  the  problem  scope. 
UR201 5,  with  all  the  advanced  sensors  available  to  the 
players,  was  still  only  automated  to  the  players  SA  Level 
1  perception.  Using  Johnson's  Criteria  and  assigning  each 
piece  of  evidence  a  mean  critical  dimension,  Equation 
1-1  can  be  used  to  generate  the  probability  of  detection, 
correct  classification,  and  identification  for  accessories 
and  rudimentary  actions,  such  as  kneeling  or  loitering. 
When  time  is  considered  (i.e.,  an  analyst  reviewing  sensor 
data  over  time),  we  may  achieve  recognition  of  behavior 
by  utilizing  the  zoom  feature  of  the  video.  It  has  been 
determined  that  to  correctly  classify  small  pieces  of 
evidence,  an  image  with  resolution  of  better  than  1  cm 
is  required  (Castleberg  et  al.,  2006). The  UR2015  sensor 
solution  required  multiple  looks  in  order  to  build  sufficient 
confidence  in  a  particularly  small  piece  of  evidence.  The 
information  provided  over  multiple  looks  was  updated 
using  Bayes'  rule,  and  only  when  the  belief  in  the  truth 
identity  of  the  evidence  was  reached  (for  UR201 5  this 
threshold  value  was  typically  set  to  0.80)  a  track  was 
generated,  containing  information  about  the  host  entity's 
location  and  velocity,  as  well  as  a  list  of  recognized  pieces 
of  evidence.  Players  then  were  left  with  the  assignment 
of  determining  if  any  given  piece  of  evidence,  or  the 
evidence  as  a  whole,  constituted  suspicious  activity. 

These  tracks,  in  addition  to  the  steaming  video,  provided 
the  players  with  the  necessary  information  to  recognize 
suspicious  behavior  and  create  SAOs  to  reflect  their  SA 
during  game  play. 
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Table  1  shows  a  listing  of  available  SAO  types,  as  well  as 
their  relative  frequency  or  appearance  during  game  play. 


Table  1 .  UR201 5  SAO  Types  and  Frequencies 


SAO  Types 

Frequency 

TerrorAct  Other 

42 

TerrorAct  Meeting 

8.5 

TerrorAct  Surveillance 

1 

TerrorAct  Suspicious 

4 

TerrorAct  Loitering 

34 

TerrorAct  Fleeing 

2 

TerrorAct  Event 

8.5 

SAO  Case  Study 

Of  critical  importance  to  today's  military  is  the  threat  of  the 
improvised  explosive  devise  (IED).  For  our  consideration, 
we  used  data  from  actual  events  from  UR201 5  constituting 
an  IED  emplacement. The  dataset  was  mined  forTerrorAct 
SAOs  where  the  players  concluded  that  an  IED  placement 
was  in  progress  or  imminent  based  on  one  of  several  pieces 
of  evidence.  We  define  an  IED  emplacement  scenario  to 
be  composed  of  the  following:  ingress  of  a  vehicle  to  a 
location  along  the  side  of  a  sparsely  populated  road,  2- 
man  team  emerges  and  loiters,  1  of  the  2  proceeds  to  the 
center  of  the  road  and  kneels  with  a  shovel,  an  IED  is  left 
behind,  individuals  proceed  to  the  car  and  mount  for  egress. 
The  process  in  sum  lasts  for  no  more  than  30  minutes.  The 
scenario  also  contains  persistent  high  resolution  imagery 
surveillance  with  one  Predator  viewing  the  area.  Figure  6,  a 
screen  capture  taken  from  the  actual  simulation  tool  used  by 
players  during  UR201 5,  illustrates  the  scenario  graphically. 
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Shovel,  Digging 
IED  Object 
Loitering  People 
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Figure  6.  UR201 5  IED  Emplacement  Scenario 


SAO  Case  Study  Results 

The  IED  emplacement  scenario  was  an  important  element 
of  the  UR2015  HITL  trial  runs,  and  players  were  trained 
on  the  indicators,  or  pieces  of  evidence,  to  look  for  to 
properly  assess  that  situation.  Figure  7  shows  the  pieces 
of  evidence  that  composed  each  event  and  the  relative 
frequency  that  each  appeared  as  part  of  the  players' 
decision-making  process.  For  example,  according  to  Figure 
7,  a  player  indicated  that  a  "hot  spot"  was  a  relevant  piece 
of  evidence  in  90%  of  SAOTerrorAct  declarations. 


Figure  7.  UR201 5  Evidence  and  Frequency  for  Determining  IED 
Emplacement  TerrorAct  SAOs 


The  UR201 5  experiment  scenarios  were  defined  to  bring 
together  many  aspects  of  behavior  and  actions.  For 
players  to  identify  a  threat,  they  would  have  to  identify 
several  pieces  of  evidence  and  correlate  them  with  each 
other.  Figures  8  and  9  show  the  various  pieces  of  evidence 
that  players  associated  with  IED  emplacement  events. 
Figure  8  shows  that,  on  average,  each  IED  emplacement 
SAO  contained  4  distinct  pieces  of  evidence. 


Figure  8.  SAO  Evidence  Counts 


Figure  9  shows  that  that  the  1 0  IED  emplacement  SAOs 
from  UR201 5  were  composed  of  1 0  distinct  pieces  of 
evidence  in  different  proportions.  This  data  shows  that,  for 
example,  most  IED  activity  happened  within  a  predefined 
"hot  spot",  in  the  presence  of  a  single  car 
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parked  next  to  a  road,  with  a  lone  individual  in  the  near 
vicinity.  The  data  showed  that  at  no  time  wasa  single  piece 
of  evidence  sufficient  to  declare  an  IED  Emplacement.  In 
fact,  on  average,  when  a  player  designated  an  SAO,  there 
were  4  nieces  of  evidence  that  contributed  to  it. 


From  SAO  to  SSAO 

As  mentioned  previously,  MCC  runs  are  able  to  use 
algorithms  focused  on  kinematics  and  features  to  determine 
rudimentary  levels  of  SA.  But  these  algorithms  are  not 
sophisticated  enough  to  evaluate  distinct  measurements 
and  group  them  together  based  on  known,  or  learned, 
patterns  to  assess  higher  levels  of  SA.  By  using  SAO  data, 
we  can  train  the  algorithms  to  watch  for  specific  pieces 
of  evidence,  each  depending  on  the  mission  or  CONOPS. 
Specifically  for  our  test  case  of  an  IED  emplacement,  we  may 
populate  Table  2  from  the  SAO  player  data. 


Table  2.  Table  of  Evidence  of  IED  Emplacement,  with  Definitions 


Categories 

Type 

Specific 

Definition 

Actions 

Loitering 

Loitering 

Individual  standing  or 
kneeling  in  roughly  same 
location  for  several  minutes 

Proximity  To  Road 

Proximity  To  Road 

Any  action,  location,  or 
information  located  at 
roadside 

Counts 

Group  Size 

Group  Size  =  1 

Observed  individual  is 
acting  alone 

Group  Size  =  2 

2  observed  individuals  close 
to  one  another 

Vehicle  count 
parked  at  roadside 

Vehicle  Count  =  1 

Observed  one  vehicle 
parked  along  roadside 

Vehicle  Count  =  2 

2  observed  vehicles  parked 
along  roadside 

Objects 

Tag 

Tag 

A  person  or  vehicle  with 
any  type  of  tag 

Object  on  Road 

IED/CI utter  object 

An  observed  object  laying 
on  the  road  (either  a 
roadside  clutter  or  IED) 

Information 

Location 

Hot  Spot 

Action  or  object  observed 
in  known  area  of  interest 

Tip 

Informant  Tip 

White  cell  injection  that 
suspicious  activity  is  taking 
place. 

IED  emplacements  in  the  real  world  vary  in  every 
dimension,  so  we  focus  on  the  essential  elements  of  an 
emplacement  to  keep  the  scenario  tractable  for  analysis.  A 
review  of  the  SAO  data  suggests  an  a  priori  emplacement 
template  for  generating  SSAOs  in  MCC  runs,  indicated  by 
Figure  9. 


Results  and  Discussion 

The  research  reported  here  is  still  in  early  developmental 
stages.  However,  we  feel  that  that  the  direction  we  are 
taking  offers  vast  potential  for  improvement  of  human 
performance  in  SA.  One  such  example  is  the  interplay 
between  sensor  development  and  human  performance 
whereby  behavior  drives  the  technological  requirements 
that  contribute  to  sensor  development.  For  example, 
looking  at  the  evidence  a  player  relies  on  is  informative 
about  the  sensor  technologies  that  are  valuable.  Being 
able  to  determine  the  physical  attributes  of  the  entities,  as 
seen  through  stealth  view,  that  were  important  to  players 
discovering  suspicious  activity  would  lead  us  to  conclude 
a  need  for  high  resolution  cameras  capable  of  detecting 
such  attributes.  When  factoring  in  the  need  to  have  these 
cameras  mounted  on  a  UAV,  aerostat,  or  towers  operating 
at  low  light  levels  and  at  night  etc.,  then  player  outcomes 
help  define  technology  requirements. 

Other  potential  applications  to  our  approach  include  the 
following: 

•  Further  refining  SAOs  to  allow  for  automation 
between  sensor  and  player  output. 

•  Training  for  SA,  such  as  by  identification  of  player 
biases. 


Concluding  Remarks 

Human-ln-The-Loop  (HITL)  experimentation  provides 
researchers  with  firsthand  data  of  how  sensors  and  sensor 
systems  are  utilized  by  the  players. Through  observation 
during  trial  executions,  researchers  and  analysts  can  watch 
the  players  while  they  make  important  time  critical  decisions 
on  how  to  improve  their  situational  awareness  through  the 
use  of  one  or  more  sensors  in  theater.  SAO  objects  are  the 
key  data  element  for  understanding  the  players' choices 
at  any  given  point  in  time.  Analyses  of  these  data  can  yield 
important  information  about  how  the  sensors  or  sensor 
systems  were  employed,  and  what  situations/scenarios  were 
the  most  useful.  Understanding  this  data  better  can 
illustrate  operational  needs  more  clearly,  which  can  thus 
affect  the  design  process  of  the  sensors  or  systems.  As  for 
systems  currently  fielded,  these  data  can  provide  insights  on 
how  to  tune  the  sensors  for  better  effectiveness  during 
varying  conditions. 
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Even  with  the  benefits  of  human  interactions  and  decision¬ 
making  during  sensor  effectiveness  studies,  due  to  the  fact 
that  HITL  experimentation  requires  a  great  deal  of  on-site 
personal  support  and  financial  resources,  Monte  Carlo 
constructive  simulations  are  an  attractive  alternative.  MCC 
runs  require  much  less  support  than  HITL  experiments  and 
are  quite  reliable  at  highlighting  the  capabilities  of  many 
sensors  and  sensor  systems  over  a  wide  range  of  conditions. 
Currently,  the  lacking  elements  of  Monte  Carlo  constructive 
simulation  runs  involve  higher  levels  of  situation  awareness, 
such  as  the  process  of  understanding  incoming  sensor  data, 
associating  tracks  based  on  this  data,  and  deducing  enemy 
intent.  By  developing  an  algorithmic  approximation  to  de¬ 
termining  SAOs  based  on  previous  HITL  data  points,  Monte 
Carlo  constructive  simulation  runs  can  achieve  a  higher  level 
of  situational  awareness  that  is  not  currently  being  obtained. 
Encapsulated  within  SAOs  are  keys  to  understanding  the 
above  mentioned  process  where  a  player  takes  sensor 
data  and  uses  it  to  update  the  overall  knowledge  of  the 
battle  space.  The  outcome  of  the  constructive  runs  can 
therefore  expand  upon  the  notion  of  sensor  and  sensor 
system  capabilities  to  include  new  areas  such  as  the 
usability  of  the  sensors  and  sensor  systems. 
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